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No one has a better chance to observe the confusion and conflicting results in 
the field of programed instruction than an editor of an educational journal. This is 
particularly true of the question of the relative efficacy of such instruction when 
compared with other means of presentation (Pressey, 1964). This report presents 
the results of an experiment involving programed learning in a typical college class- 
room, designed to study the relationship between content dimensions (both relevant 
and irrelevant) and method of presentation (teaching machine and programed text- 
book). The objectives of the research were two-fold: (a) To develop a procedure for 
the use of automated instructional devices and programed learning to supplement 
the methods traditionally used in the teaching of an introductory psychology course 
for prospective public school teachers; and, (b) To evaluate the learning outcomes 
resulting from the use of programed text material which is highly relevant and a 
highly relevant program presented by a Skinner-type teaching machine, as opposed 
to learning outcomes resulting from the use of programed text material which is 
only slightly relevant and a slightly relevant program presented by a Skinner-type 
teaching machine. 


PROCEDURE 


Subjects. Subjects for the study consisted of male and female undergraduates at Northwestern 
University who were enrolled in a two-quarter course in introductory psychology for education majors. 
The course was taught and administered joint] by the School of Education and the Department of 
Psychology. Out of an original enrollment of 130 students, 114 subjects remained in the course 
throughout the two quarters and constituted the total sample for the research. 


Treatments. At the beginning of the Fall Quarter 1963, the students were divided equally and 
randomly into five groups. Each group was further randomly assigned to an experimental treatment 
or to a control condition for the duration of the experiment. Four of these groups were exposed for 
the two quarters to different experimental treatments involving the use of highly relevant and slightly 
relevant materials presented through the use of a teaching machine and a programed textbook. The 
fifth group was given only the traditional lecture and textbook materials. 

The control group received the regular lectures plus assignments in E. R, Hilgard’s Introduction 
to Psychology, 3rd ed. (1962), and further assigned readings from The Scientific American. 

The two experimental groups subjected to the “highly relevant’’ condition were required in addi- 
tion to the above to go through the programed workbook or Student Guide prepared by Teevan and 
Jandron (1962) for use with Hilgard’s text. It consists of 1,350 frames of the constructed response 
type with no branching. One group used it as a programed text. For the other group, the 1,350 frames 
were adapted for automation by being put on a continuous tape for use in the Rheem Califone Didak 
501 machine. 

The “slightly relevant’’ conditions utilized the programed text by James G. Holland and B. F. 
Skinner, The Analysis of Behavior (1961). This is a text consisting of 2,000 frames which presents a 
Skinnerian point-of-view in introductory psychology. For one group it was used in its original form as 
a programed text, for the other it was put on a tape for use in the Didak machine. 

The regular lectures and discussions were given to the entire class during the scheduled class 
meetings. Those students having assignments in programed text or on teaching machines executed 
them in addition to the regular class hours. These experimental groups were allowed to set their own 
pace with the programed materials. 

In summary, the randomly constituted groups received the following treatments: 

Group A: Holland and Skinner programed text in addition to regular lectures, the Hilgard text- 

book, and assigned readings. 

Group B: Holland and Skinner programed text materials presented through a Rheem Califone 

ics 501 Machine in addition to regular lectures, the Hilgard textbook, and assigned 
readings. 


1This project was done under the sponsorship of Title VII of the National Defense Education Act 
(Project No. 1075). 
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Group C: The Teevan and Jandron programed workbook for the Hilgard text in addition to the 
Hilgard text, regular lectures, and assigned reading. 


Group D: The Teevan and Jandron programed materials for the Hilgard text presented through 
a Rheem Califone Didak 501 Machine in addition to regular lectures, the Hilgard 
text, and assigned readings. 


Group E: Regular lectures and the Hilgard text plus assigned readings. 


Criteria for Performance Change. At the beginning of the Fall Quarter, before the start of class 
lectures, the total class was given a test of psychological knowledge consisting of random items picked 
from the multiple-choice items available for the Hilgard text, Introduction to Psychology (1962). These 
items were obtained from an item pool available to accompany the Hilgard text. Each chapter of the 
text is represented by twenty multiple-choice items. From this item pool, a balanced number of items 
were randomly picked from each chapter so that a random item test of 230 multiple-choice items was 
assembled. This pretest was administered on two successive days with 115 items presented to the total 
class on the first day and 115 items on the second day. In addition, during the Fall Quarter, two 50 
item examinations and one 100 item examination consisting of multiple-choice items chosen from the 
same Hilgard item pool, but avoiding the items used in the pretest, were administered. These were 
given at appropriate intervals so that each test covered approximately one-third of the course and did 
not include material previously tested. The last of these three tests, the 100 item examination, was 
given during the final examination period for the Fall Quarter. 

Except for the pretest the same testing procedure was followed during the Winter Quarter, with 
three multiple-choice tests being administered, consisting of the same number of items as during the 
Fall Quarter, covering approximately one-third of the course at each testing. The lectures and Hilgard 
textbook assignments were arranged so that approximately the first half of the Hilgard text was cover- 
ed during the Fall Quarter and the last half during the Winter Quarter. 

At the end of the Winter Quarter during the last week of class, and following the final 100 item 
test, the pretest consisting of 230 randomly selected Hilgard items was again administered in a two- 
day sequence with 115 items being given the first day and 115 the second. This posttest was identical 
to the pretest given at the beginning of the course. 


Instructions Given to the Groups. At the beginning of the Fall Quarter the instructors explained to 
the students that, in addition to the instruction they would receive in the course, they would be in- 
volved in a research project which would allow them to gain some insight into experimental procedures 
in general and the process of experimentation in learning. They were told that they would be divided 
into various groupings, but that the groupings would have no influence on the grade they would re- 
ceive in the course. They were also told that some groups would be given various tasks to complete 
and that their selection for a particular group was based on chance aad not on any abilities they might 
or might not possess. The point that group assignment would not influence their grade was stressed 
throughout the course. The instructions were such that the investigators felt that the students were 
motivated to participate and respond. At the same time the instructions were not so detailed as to 
appraise the students of the specific purposes of the experiment. 

Each of the four experimental groups (A, B, C, D) was given additional detailed instructions 
concerning the use of the teaching machines and the programed texts. A capable student assistant was 
in charge of scheduling the use of the machines. He was present and svailable during both quarters at 
specific times during the week in an automated instruction laboratory in the basement of a campus 
building easily accessible to the students. The students in the teaching machine groups (B and D) 
were allowed to set their own pace in scheduling appointments, but they were expected to complete an 
assigned number of units of each program during one quarter. The students in groups A and C were 
issued copies of the appropriate programed text material and were encouraged to use them and to 
respond to the items by writing in the books. 

At the beginning of the Fall Quarter the five groups consisted of approximately 27 students each. 
However, normal attrition was such that the analysis of results was based on the following number 
of subjects in each group: Group A - 22, Group B - 23, Group C - 23, Group D - 23, Group E - 23. 


REsvULtTs 


The analysis of data was accomplished by an analysis of covariance procedure 
using scores of the pre- and post-achievement test as covariant and criterion in one 
type of analysis, and pretest achievement scores and total examination points for 
the Winter Quarter as covariant and criterion in another type of analysis. 

In all of the analyses? scores on the 230 item test of relevant psychological 
knowledge given at the beginning of the Fall Quarter were used as covariants. The 


"Tables have been deposited with the American Documentation Institute. Order Document No. 
8594, remitting $1.25 for 35 mm. microfilm or $1.25 for 6 by 8 in. photocopies. Advance payment 
is required. Make checks or money orders payable to: Chief, Photoduplication Service, Library of 
Congress. 
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criteria for achievement, or changes in performance, were either the scores on the 
230 item test given at the end of the course or the total number of examination 
points obtained during the Winter Quarter from the three examinations given during 
that quarter. Winter Quarter examination points were used because an analysis of 
Fall Quarter examination points indicated no significant differences between condi- 
tions. In addition the machine groups were not able to start their experimental 
trials until two weeks after the beginning of the Fall Quarter, and it was felt that 
the cumulative effects of changes in achievement would be more pronounced during 
the Winter Quarter. 

No significant differences were obtained, using the posttest achievement scores 
and the examination points as criteria, in a comparison of all five groups; although 
evidence of a trend in mean scores favoring the experimental groups did appear. 
Differences were obtained, however, at greatly beyond the .01 level of significance in 
comparing pre- and posttest scores for the class as a whole. 

Since no attempt was made to control the use of the programed text by experi- 
mental groups A and C, other than to impress on the students the necessity and 
importance of using them for purposes of the experiment, an assessment of their 
use was made at the end of the course. A majority of students in group A indicated 
that they used the Hollard-Skinner programed text infrequently, while a majority 
of students in group C indicated a use of the Teevan-Jandron programed material 
most of the time. For this reason, an analysis was made combining group A with 
experimental group E and testing for differences between four groups instead of 
five. No significant differences were obtained, although the data did indicate a 
trend in means in favor of the experimental groups. 

Because of this trend, it was decided to treat the experimental groups B, C, and 
D as one group, and to compare this larger experimental group to the already com- 
bined groups A and E which made up the control group for the comparison. The 
increase in the number of subjects in the groups was thought to offer a better basis 
of ascertaining significant differences between means where trends were already 
evident. These two groups did differ significantly (at between the .05 and .01 level 
of confidence) in the analysis of pre- and posttest scores on the 230 item test of 
relevant psychological knowledge, and they differed significantly (at between the 
.05 and .01 level of confidence) in the analysis of pretest and total Winter Quarter 
examination points. 


Discussion 

In discussing the results it might not be out of order to indicate that the gains 
in scores between the pre- and posttest of relevant psychological information does 
demonstrate that it is possible to teach the content of the Hilgard text in an intro- 
ductory course in psychology. At the risk of doing the authors an injustice it also 
seems possible to infer that the students were not attracted by the Holland-Skinner 
text, remembering that when it was put on a tape for machine use and students 
were required to schedule appointments, it seemed to have more attention value. 

In general, our results resemble those of other ‘‘shotgun” type experiments in 
the classroom in which the significant variables are difficult to define and are ob- 
scured by the small size of samples used. At the risk of proceeding with only minimal 
justification, the experimenters would like to offer their explanation of the difference 
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that does appear when the three effective experimental conditions are treated as one 
and the control condition is combined with the ineffective experimental condition 
(the non-use by the students of the Holland and Skinner text), although there are 
no differences between the four experimental conditions themselves. Following 
Underwood (1959) we would suggest that the single most important variable in 
learning is the time spent in learning and that the outstanding difference between our 
effective experimental conditions and the control conditions was that the experi- 
mental conditions assured an added time involvement by the students in the learn- 
ing process. The implications of this for education are obvious. 
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Studies reporting the superiority of immediate knowledge of results over delayed 
feedback have appeared with some regularity during the past sixty years (Ammons, 
1956; Landsman, 1962). Most of the research on knowledge of results has been 
concerned with non-meaningful verbal materials or motor skills. In the last few 
years, studies (Brackbill, Bravos, & Starr, 1962; Brackbill, Isaacs, & Smelkinson, 
1962; Brackbill & Kappy, 1962; Brackbill, Wagner, & Wilson, 1964) have been 
reported which throw some doubt upon the universality of the principle of immedi- 
ate knowledge of results when one considers the type of learning and the different 
organisms involved. Inferences based on studies of motor tasks do not necessarily 
yield information about verbal tasks, especially when the verbal tasks are meaning- 
ful. 

The investigators during the past two years have participated in research pro- 
jects studying the effectiveness of adjunct auto-instruction providing immediate 
knowledge of results (Cooperative Research Project No. 2306). Discussions of the 
research staff during this period raised the issue of whether immediate knowledge of 
results is necessary for the learning of meaningful verbal materials. The present 
study compares immediate knowledge of results with feedback delays of one hour, 
two days, and one week. 


1This research was supported in part by the Cooperative Research Program of the Office of 
Education, United States Department of Health, Education, and Welfare, Project Number 2761. 
"Now at the University of Missouri. 


